Repository navigation
chore: bump kubedog to fail fast on ReplicaSet pod-create errors - #780
Merged
Merged
Conversation
Signed-off-by: Ilya Lesikov <ilya@lesikov.com>
Signed-off-by: Ilya Lesikov <ilya@lesikov.com>
This was referenced Oct 30, 2026
Merged
dmmordvi
added a commit
that referenced
this pull request
Nov 3, 2026
## Summary Bumps `github.com/werf/kubedog` to the merge commit of werf/kubedog#406 on kubedog `main` (`1ed920d`). A long-lived process that runs `nelm` release actions over and over with tracking (for example, a controller that deploys a release on every reconcile) no longer accumulates informers that hold the cache of a resource type for a whole namespace after each action ends. Before, the memory such a process held grew with every deploy until it restarted. ## What - `go.mod`/`go.sum`: `github.com/werf/kubedog` `v0.13.1-0.20261029011008-97f4f131413e` → `v0.13.1-0.20261103160714-1ed920dee18b`. The range `97f4f13..1ed920d` is the single commit of werf/kubedog#406. - After a release install, upgrade, uninstall or rollback with tracking finishes and its context is canceled, the shared informers it started stop and release their caches, even when the trackers stopped reading before an informer delivered all its events. Applies to the Pod, Deployment, StatefulSet, DaemonSet, Job, Canary and ReplicaSet trackers, the generic resource state and events watchers and the event informer that `pkg/dyntracker` uses. - A fatal watch error raised after the action finished no longer blocks the informer: it is dropped once the informer factory is stopped instead of waiting forever on the watch error channel. - When a Job, DaemonSet, StatefulSet or Pod tracker fails halfway through starting, the informers it already started are released. - While tracking runs, every event is delivered as before: channel capacities and send order are unchanged, nothing is dropped. Tracking results, progress output and errors do not change. - Library API: kubedog adds the exported helper `utils.SendOrDone(ctx, ch, v)`. Nothing in the `nelm` API changes. - `go.mod`: `golang.org/x/sync v0.20.0` moves from the indirect to the direct block, same version. `pkg/resource/spec/patches_debug_test.go` (#785) already imports it directly; `go mod tidy` on `main` without this bump makes the same change. - UNVERIFIED: not reproduced against a real cluster. kubedog covers every changed handler with a regression test on a fake dynamic client that shows the informer goroutines stay alive without the fix and exit with it. ## Why Informer event handlers run in the client-go listener goroutine and sent to tracker channels without watching for cancellation. When a handler had more events to deliver than the channel could buffer at the moment the tracker stopped reading (e.g. the initial Add events for several Pods of a Deployment that is already ready), the send blocked forever, the informer stayed in `wg.Wait()` after the stop signal, and its indexer stayed referenced. 38643e9 in kubedog (brought in by #780) fixed this for the ReplicaSet informer only. Bigger channel buffers only move the threshold, since the number of objects an informer replays on start is unbounded. Signed-off-by: Dmitry Mordvinov <dmitry.mordvinov@flant.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Bumps
github.com/werf/kubedogto the merge commit of werf/kubedog#404 on kubedogmain(97f4f13), which makes Deployment tracking fail fast on ReplicaSet pod-create errors such asexceeded quota(werf/kubedog#398).werf/kubedog#404 is merged; this PR pins its merge commit.
What
go.mod/go.sum:github.com/werf/kubedogv0.13.1-0.20260915153156-438ef109f804→v0.13.1-0.20261029011008-97f4f131413e.maincommits since the previous pin (438ef10..b201189): golangci-lint cleanups and reformatting. They do not change behaviour.FailedCreatemessage are bounded, so a crafted ReplicaSet status cannot keep the tracker busy.Why
Without this bump,
nelm(andwerf) keep waiting for the tracking timeout when a Deployment's new ReplicaSet cannot create pods. The failure is only visible in ReplicaSet events.Behaviour changes for nelm users
These come from
pkg/dyntracker, whichpkg/plan/plan_execute.gouses:werf.io/failures-allowed-per-replica. A quota deadlock is reported about 10s after it is observed, with theFailedCreate ... exceeded quotadiagnostic.werf.io/fail-mode: IgnoreAndContinueDeployProcess, these fatal failures do not fail the deploy.list pods(already needed for tracking) andget resourcequotasin the namespace. The built-inview,editandadminroles have both. Without them, quota failures are counted rather than fatal.Testing
task build,task test:unit(67 suites) andtask lintpass.ResourceQuota, tracking throughdyntrackeras nelm does.